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ABSTRACT 


A model of human pilot performance of a tracking task using a generic 
Cockpit Display of Traffic Information (CDTI) display is developed from 
experimental data. The tracking task is to use CDTI in tracking a leading 
aircraft at a nominal separation of three nautical miles over a prescribed 
trajectory in space. The analysis of the data resulting from a factorial 
design of experiments reveals that the tracking task performance depends on 
the pilot and his experience at performing the task. Performance was not 
strongly affected by the type of control system used (velocity vector control 
wheel steering versus 3D automatic flight path guidance and control). The 
model that is developed and verified results in state trajectories whose 
difference from the experimental state trajectories is small compared to the 
variation due to the pilot and experience factors. 
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INTRODUCTION 


The Advanced Transport Operating Systems (ATOPS) program (formerly called 
the Terminal Configured Vehicle (TCV) program, ref. 1) has been established by 
NASA to perform flight management and operating systems research broadly aimed 
at improving the safety and efficiency of transport aircraft all-weather 
Operations in the evolving National Airspace System. The goal of the ATOPS 
program is to blend recent and emerging technology advancements in airborne 
avionics systems and information displays with human factors into effective 
system concepts that cart be Uniformly applied to transport aircraft operating 
in a 1 990 ' s air traffic environment. Specific objectives of the ATOPS program 
are to propose and investigate concepts offering improvements to aircraft 
systems, flight deck design, and crew procedures providing more efficient 
operations; to develop and investigate ways to improve the exchange of 
information between aircraft and air traffic control (ATC) throughout the 
flight profile; and to identify and promote consideration of aircraft 
capabilities and limitations in the design of ATC system improvements to 
facilitate more efficient operations. These are accomplished by conducting 
analysis, simulation, and flight tests and by sponsoring similar research by 
the aircraft industry. 

A major research facility available to the ATOPS program is the Mission 
Oriented Terminal Area Simulation (MOTAS) facility (ref. 2). This facility is 
designed to provide a highly realistic environment in which air transport 
crews and air traffic controllers can participate in the conduct of flight 
management artd flight operations research studies. The goal of the study 
described in this report was to develop an algorithm which could be used in 
the Mission Oriented Terminal Area Mathematical Model (MOTAM) to model an 
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aircraft equipped with a generic CDTI display. This class of aircraft would 
be included with area navigation ( RNAV ) , VOR/DME, MLS, ILS, etc. equipped 
aircraft in the model. 

One type of CDTI display would enable a pilot to observe the locations of 
nearby aircraft on a map display in the cockpit. This information is useful 
when the visibility is poor and the aircraft is flying under instrument flight 
rules (IFR). Under IFR, the ground controller gives directions to the pilot 
which ordinarily keep the aircraft a safe distance from others in the area. 

As an added precaution, the minimal allowable separations between aircraft are 
increased under IFR. Unfortunately, this increases the minimum time required 
between successive aircraft landings and thus tends to result in a lower 
airport throughput rate. Thus the inability to see aircraft under IFR can 
result in slower landing service to incoming aircraft. CDTI enables a pilot 
to observe the locations of nearby aircraft, resulting in the possibility of 
using a modification of IFR rules for aircraft having CDTI capability. This 
could result in faster landing service in bad weather. In addition, CDTI 
offers some possible advantages regarding air traffic safety and collision 
avoidance when the visibility is good and the aircraft is flying under visual 
flight rules (VFR). Under VFR, the CDTI would allow the pilot to see the 
locations of aircraft not normally visible through the windscreen of the 
cockpit due to range or position conditions. 

Some changes in the air traffic control rules will be required to realize 
the potential benefits from CDTI. One of the difficulties of designing a new 
air traffic control system is the problem of predicting performance under the 
new system. It is of course economically infeasible to engage a fleet of 
aircraft and an airport for weeks or months to study a new system. Simulation 
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has evolved as a primary means of evaluating system performance. The Terminal 
Area Air Traffic Model (TAATM) was developed to model the flow of air traffic 
in the vicinity of an airport terminal area. The model simulates air traffic 
flow by the generation of information describing the flight of aircraft 
arrivals and departures. In past studies, it has included terminal geometry 
corresponding to either of two airports (Atlanta or Denver) and has been used 
to study various aspects of ATC system performance. 

An additional degree of modeling and realism was obtained by interfacing 
human pilots in a simulator cockpit with the TAATM model (refs. 3 and 4). 

This became the prototype for the concept of the Mission Oriented Terminal 
Area Simulation which allows for human pilots flying simulator cockpits to be 
substituted for computer-generated aircraft in the terminal area model. The 
result is a model of the terminal area environment (ref. 5) with human pilots 
and air traffic controllers interacting with simulated pilots and controllers 
to simulate air traffic flow. This simulation requires the use of models of 
human pilot and controller performance of various tasks. 

A series of experiments requiring human pilots to use a generic CDTI 
display in tracking a leading aircraft were performed. One of the objectives 
of the experiments was to log data that could be used to identify parameters 
in a model describing human performance of the tracking task. The model may 
then be applied in a MOTAS-type of simulation involving human pilots in 
simulator cockpits interacting with each other and the models of CDTI-equipped 
aircraft assigned tracking tasks. 
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SYMBOLS 


A edj - experimental average separation between lead aircraft and 

pilot's aircraft for jth region, nmi 

A ev j experimental average groundspeed of the pilot's aircraft 

for jth region, kts 


A mdj 


modeled average separation between lead aircraft and 
pilot's aircraft for jth region, nmi 


A mvj 


modeled average groundspeed of the pilot's aircraft for 
jth region, kts 


time integral of the squared position error between the 
experimental and model trajectories 


E y time integral of the squared velocity error between the 

experimental and model trajectories 

E weighted sum (WE + W v E y ) of position and velocity 

errors 1 


H nominal separation between the lead aircraft and the 

pilot's aircraft, nmi 

M( ), S( ) denotes an operator which yields a sample mean and sample 

standard deviation, repectively 

P a vector of model parameters where the parameters are 

? j w > H, x 

R ed - experimental RMS deviation of the separation between the 

lead aircraft and the pilot's aircraft for the jth region 



experimental RMS deviation of the groundspeed for the 
pilot's aircraft for the jth region 


R |nd j modeled RMS deviation of the separation between the lead 

aircraft and the pilot's aircraft for the jth region 

R mv j modeled RMS deviation of the groundspeed for the pilot's 

J aircraft for the jth region 


W vector of weights applied to Ep and E y . W = (Wp, W v ), 

where 0 <_ W p <_ 1 and W p + W y = 1 

X separation error, X = (Z Q + H) - Z^, nmi 
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X 

X e (t), x e (t) 

X m (t; S,u>,H,t) 
and 

X m (t;s,u),H,x) 

Z 0 

• • • 
z 0’ z 0 

Z 1 

* • » 

Z 1 * Z 1 

C, 0) 

T 

superscripts 
T, -1 


velocity difference (Z Q - Z^, nmi/sec 

state variables at time t from an experimental run 

state variables at time t from a model with parameters 

C, 03 , H, T 

distance from origin to the pilot's aircraft, nmi 
velocity and acceleration of pilot's aircraft, respectively, 

O 

nmi/sec, nmi/sec c 

distance from origin to lead aircraft, nmi 

velocity and acceleration of lead aircraft, respectively, 

O 

nmi/sec, nmi/sec 

parameters in differential equation model 

delay in feeding back position and velocity errors, sec 

transpose and inverse, respectively 
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ACRONYMS 


AGCS 

ATC 

ATOPS 

CDTI 

CRT 

DME 

EADI 

EHSI 

IFR 

ILS 

MLS 

MOTAM 

MOTAS 

NCDU 

RMS 

RNAY 

TAATM 

TCV 

TSRV 

VCWS 

VFR 

VHF 

VOR 

3D 


Advanced Guidance and Control System 

Air Traffic Control 

Advanced Transport Operating Systems 

Cockpit Display of Traffic Information 

Cathode Ray Tube 

Distance Measuring Equipment 

Electronic Attitude Director Indicator 

Electronic Horizontal Situation Indicator 

Instrument Flight Rules 

Instrument Landing System 

Microwave Landing System 

Mission Oriented Terminal Area Model 

Mission Oriented Terminal Area Simulation 

Navigation Control and Display Unit 

Root Mean Square 

Area Navigation 

Terminal Area Air Traffic Model 
Terminal Configured Vehicle 
Transport Systems Research Vehicle 
Velocity Vector Control Wheel Steering 
Visual Flight Rules 
Very High Frequency 
VHF Omnidirectional Range 
3-dimensional 
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MOTAS FACILITY DESCRIPTION 


MOTAS is a flexible, comprehensive simulation of the airborne, ground- 
based, and communications aspects of the terminal area environment. The major 
elements of MOTAS (fig. 1) are: a terminal area environment model, aircraft 

simulator cockpits, pseudo pilot stations, air traffic controller stations, 
and a realistic air-ground communications network. The terminal area 
environment model (ref. 5) represents today's Denver Stapleton International 
Airport arid surrounding area using either an automated metering and spacing 
system of control or a manual vectoring system of control. In addition, the 
model simulates various radar systems, navigational aids, wind conditions, and 
so forth. 

The facility is configured to tie-in a number of aircraft simulator 
cockpits including the Transport Systems Research Vehicle (TSRV) Simulator, 
the DC-9 Full Workload Simulator, the Advanced Concepts Simulator, and the 
General Aviation Simulator. This allows for a number of different types of 
aircraft to be flown in a realistic environment by human crews. The pseudo 
pilot stations (fig. 2) are devices which allow an operator to control and fly 
a number of computer-generated aircraft in the terminal area environment along 
with the aircraft simulator cockpits, thus providing the realism of the 
traffic density which would occur in the real world. The pseudo pilot 
stations operate through the use of voice recognition technology. 

The air traffic controller stations (fig. 3) are composed of advanced 
computer graphics systems and host computers to provide the controller with 
generic, but realistic, displays. The displays can be programmed to represent 
present day display formats or to represent futuristic formats for research 
purposes. Finally, a realistic air-ground communications system is simulated 
to provide the real world communications environment. 
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TSRV SIMULATION DESCRIPTION 


The ATOPS Program operates a Boeing 737-100 aircraft (fig. 4) to conduct 
flight research aspects of the program. The aircraft is equipped with a 
special research flight deck, located approximately 20 feet aft of the 
standard flight deck. An extensive array of electronic equipment and data 
recording systems is installed throughout the former passenger cabin 
(fig. 5). The aircraft can be flown from the aft flight deck research cockpit 
using advanced flight control and electronic display systems that can be 
programmed for research purposes. Two safety pilots located in the standard 
flight deck are responsible for all phases of flight safety and for most 
traffic clearances. Two research pilots usually fly the aircraft from the 
research cockpit during test periods, which can last from takeoff through 
landing. 

Figure 6 presents a picture of the interior of the TSRV Simulator, which 
is a near replicate of the research cockpit located onboard the TSRV research 
aircraft. The simulation includes a nonlinear mathematical model of the 
aircraft with the addition of landing gear dynamics, gust and wind models, 
nonlinear actuator models, and instrument and microwave landing system (ILS 
and MLS) sensor models. In addition, automatic flight control and navigation 
control functions have also been simulated. The simulator cockpit is 
outfitted with advanced flight control and electronic display systems. These 
include an advanced guidance and control system (AGCS), an electronic attitude 
director indicator (EADI), an electronic horizontal situation indicator 
(EHSI ) , and a navigation control and display unit (NCDU). 

The various AGCS modes are engaged using the control panel pictured in 
figure 7. These control modes provide the pilot with desired levels of 
automation and are designed to relieve the pilot's workload. The AGCS modes 
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include two levels of control wheel steering (attitude and velocity vector), 
four levels of outer loop guidance and control (track angle and flight path 
angle select, horizontal path guidance, vertical path guidance, and time path 
guidance), an autoland system, an altitude hold system, and an autothrottle 
system. 

The EADI, pictured in figure 8 and described in reference 6, is the 
pilot's primary display of pitch and roll attitude for instrument flight. 
Optional symbology for display of the aircraft velocity vector, flight path 
acceleration, vertical and horizontal guidance errors, speed error, 
perspective runway and centerline, and radar altitude are integrated into the 
EADI display format. 

The EHSI, pictured in figure 9 and described in reference 7, is the 
pilot's primary navigation display for instrument flight. It is configured to 
represent a map and provides the pilot with an accurate display of aircraft 
position relative to the horizontal guidance path, flight plan waypoints, and 
geographic points of interest. The desired horizontal flight plan is 
displayed by a solid line connecting the waypoints. The operating modes of 
the pilot's and copilot's EHSIs are independent and may be operated in either 
track-up (normal) or north-up modes and with different scales and different 
information options. 

The primary input device to the navigation and guidance system is the 
NCDU pictured in figure 10. The unit consists of a keyboard and a small 
cathode ray tube (CRT) display on which pages of navigation and guidance 
information may be displayed. Guidance paths may be built using the NCDU as 
an input/output device. During the flight, variables of interest, such as 
path guidance errors, may be displayed on the CRT. 
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ANALYSIS 


Modeling Human Performance of the Tracking Task 

The experimental data has been analyzed from the standpoint of a one- 
dimensional tracking problem. The state variables can be regarded to be the 
distance traveled along a prescribed trajectory in space and its time 
derivative. Alternatively, the position error and its time rate of change can 
be the state variables, since their knowledge used with certain known 
information allows the distance traveled of the pilot's aircraft and its 
derivative to be calculated. It is assumed that other control "systems" 
(including pilot control actions that are not being modeled) keep the aircraft 
on the desired trajectory in space. The tracking task is to maintain a 
nominal spacing of 3 n. mi. between the pilot's aircraft and the lead aircraft 
by the use of speed control only. 

In each of the forty flights, a pilot performed speed control on his 
aircraft in consideration of many factors, including the flight plan and his 
perception of the separation and its rate of change between his aircraft and 
the lead aircraft. Each experiment resulted in a state trajectory which was 
sampled and recorded in Regions 2 and 3 of the flight plan. The modeling 
problem considered herein for each flight was to find a mathematical model 
driven by the state variable feedback whose state trajectory approximated the 
state trajectory from human pilot performance. 

The model 

Z Q (t) = -2^X(t-x) - w 2 X(t-x) 

was selected after some experimentation to model performance of the tracking 
task. In this equation, C is the damping coefficient, u> is the undamped 
natural frequency, and x is the delay in feeding back the state of the 
system. Since the definition 
X = (Z 0 + H) - Z X 
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includes the nominal separation H as a parameter, the acceleration of the 

pilot's aircraft given by the model depends on the parameter vector (£, u ,H,t). 

Identification Problem 

The measure Ep of the position error between an experimental trajectory 
and a model trajectory was defined to be 

E p (?,aj,H,T) = /(X e (t) - X m { t; c.oo.H.t) ) 2 dt. 

Similarly, the velocity error was defined as 

E v U,oj,H,t) = /(X e ( t) - X m ( t; c.oi.H.t) ) 2 dt. 

A weighted sum of the position and velocity errors was used as the 
overall error E{ ?,w,H,T;W p ,W v ) : 

EU,^H,T; Wp ,W v ) = W p E p U,co,H,T) + W v E v U,oj,H,t). 

With these measures of modeling errors, the identification problem 
becomes 

Min E(c ,( d,H,t;W ,W ) or equivalently 

U.w.H.t) P 

Min {w d E_U,w,H,t) + W v E v U,w,H,t)L 

which depends on the weights W p and W v that are used. The weight W p is 
applied to the square of the difference between the position error X m 
predicted by the model and the actual position error X g from the experiment. 
Similarly, the weight W y is applied to the square of the velocity error 
difference between model and experiment. Thus the weights (W p ,W y ) = (1,0) 
give an error that does not explicitly depend on the velocity error match 
between the model and experimental data. Also, (W p ,W v ) = (0,1) results in no 
explicit dependency of the measure of goodness of fit of the position error 
match between the model and the experimental data, while (W p ,W y ) = (0.5, 0.5) 
gives some weight to matching both position error and velocity error. There 
may be room for argument regarding the "best" weights to use in the integral 
square error measurement. The analysis described herein was done using the 
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weights Wp = W y = 0.5. The resulting family of model trajectories approximate 
their experimental counterparts with errors that are acceptably small for many 
areas of model application. Of course, the judgement as to whether or not the 
accuracy is acceptable can be made only in consideration of the context of 
application of the model. If smaller modeling errors are needed for a 
specific application, the weights Wp and W y can be changed to obtain new 
families of model trajectories. 

When the weights W p and W v are fixed (the values W p = W y = 0.5 were 
used), an identification problem involving the four parameters c. to, H, and x 
results. If computational time were not a consideration, one could search 
over the four parameters to find the parameter vector which yielded the 
minimum value of the error E( C,w,H,x,Wp,W y ) . Unfortunately, the system must 
be simulated each time the error is to be evaluated for a given parameter 

vector (c,uj,H,t). Consequently, an approximate solution to the identification 

problem was sought by using the procedure: 

(i) using a heuristic search in (H,x) space to find promising values 

of H and x. The equation error method was used to calculate 

C, a) and the attendant error for given values of H and x. 

( i i ) searching in U,to) space for given values of H and x to find a 
local minimum value of the integral squared error. 

(iii) repeating steps (i) and (ii) until a reasonably good fit was 
obtained. 

This procedure, which decomposes the search in four dimensions into two 
searches, each in two dimensions, was found to result in acceptable 
computational times. The procedure was arrived at after performing a series 
of numerical experiments to investigate the tradeoff between computer time and 
modeling error. 

The equation error method was applied to the problem of selecting H 
and x as follows. When H and x are fixed, the pilot's aircraft acceleration, 
Zq, predicted by the model depends only on ? and to. At time t n , we have the 
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pilot's aircraft acceleration observed from the experiment and the pilot's 
aircraft acceleration predicted from the model: 

Z 0 (t n ) = -2£wX(t n -T) - a) 2 X(t n -x). 

Thus when exactly two observations (at distinct times t m and t n ) are used, we 
have two equations and two unknowns, and the values of c and to can be 
calculated provided that the resulting equations are independent. When more 
than two observations are used, there results an overspecified system of 
equations. For example, if there were exactly three distinct observations 
(taken at times t 1# t 2 , and t 3 ), the system of equations is 


J- C_ _ 

-2X(t r x) -X(t r x) 


?to 


z o 

-2X( t 2 -x) -X(t 2 -x) 


(O 2 

= 

z 0 (t 2 ) 

-2X( t 3 - x) -X(t 3 -x) 



_ Z 0 ^3* 


Assuming the observations are independent, an overspecified system of 
equations of the form Au = b results, where 


A = 

”-2X(t 1 -r) -X(t r x) 

, u = 

C*o 

, and b = 

z 0 ^1^ 


-2X(t 2 -x) -x(t 2 -x) 


to 2 


Z 0 (t 2^ 


-2X(t 3 -x) -X(t 3 -x) 




. Z 0 


The overspecified system of linear equations represented by Au = b can be 
"solved" in the least squared error sense by u = (A^A)"^A^b. The resulting 
solution u minimizes the sum of the squared errors associated with each 
equation. The squared error is b T b-(A^b)^(A^A)“^(A^b) . 

In summary, the equation error method enables c and to to be calculated 
analytically when W, H and t are fixed, as opposed to having a search over 
(c,(o) space done with the evaluation of each point in (?,<o) space requiring a 
time integral to be done to calculate its attendant error. Consequently, a 
two-dimensional search over H and t can be done to find starting values for 
H, t, ?, and to. 
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When W, H, and t are fixed, the identification problem simplifies to 



where Ep (£,w;H,t) is the time integral of the square of the difference in the 
position errors between the experimental and model results, and E v (C,^;H,t) 
is defined similarly for the velocity difference. 

The values of K and w that result in a minimum value of the integral 
squared error E(s,u),H,r,Wp, W y ) for H and t given were sought as follows. First 
of all, the equation error method was used to give initial values for K 
and u) . Then the value of the integral squared error E(?,w,H,T,Wp,W v ) for H 
and t given was calculated by simulating the performance of the model 
represented by the parameters (c,co;H,t) in Regions 2 and 3. This simulation 
required the use of numerical integration. The triangular rule with a time 
increment of four seconds was found to yield satisfactory results. The point 
(c,oj) was accepted as a local minimum when the following conditions were met 
E(?+Ac,o);H,t ,W p ,W v ) > E(c,aj;H,T,Wp,W v ) - DE 
E(C-Ae,<o;H,T,W p ,W v ) > EU,io;H,x,W p ,W v ) - DE 
E( c,w+A(jo;H,T,Wp,W v ) > E( c,w;H,T,W p ,W v ) - DE 
E( C,o)-Aw;H,T,Wp,W v ) > E( S.w ;H, r,W p ,W y ) - DE 
that is, the new point must be better than the old point by more than DE (a 
fixed number) to be regarded as yielding a significantly better fit. When a 
new best point is found, it becomes the starting point for another similar 
search in the four directions. The process is continued until a local minimum 
is found. The parameter DE was determined experimentally by trying different 
values and observing their impact on the attendant modeling errors. For the 
triangular method of numerical integration with a time increment of four 
seconds, the value DE = 0.20 was found to yield satisfactory results. 
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DESCRIPTION OF EXPERIMENTS 


A factorial design of experiments was used with four pilots, two control 
systems, and five replications for a total of forty experiments. Four 
separate pilots were used in anticipation of task performance dependency on 
the person performing the task. The two control systems - velocity vector 
control wheel steering (VCWS) and 3D automatic flight path guidance and 
control both coupled with manual throttles - result in different work loads. 
The VCWS requires the pilot to perform certain control tasks related to 
maintaining his aircraft on the desired trajectory in space which are done 
automatically in the 3D control mode. The workload under VCWS could 
conceivably be detrimental if it prohibits the pilot from giving enough 
attention to the tracking task. On the other hand the light workload under 3D 
could be detrimental if it is insufficiently demanding to hold the pilot's 
concentration. Five replications were used for each combination of pilot and 
control system to allow measurement of variation within each cell. 

For a given test run, the pilot was instructed to try to maintain a 
separation of 3 n. mi. between his aircraft and the lead aircraft. This was 
to be accomplished primarily through speed variation by use of the throttles, 
speed brakes, and flaps. Both aircraft used the same flight plan, and it was 
displayed to the pilot on the EHSI. Since the flight plan (fig. 11) involved 
a constant speed and altitude segment followed by a deceleration segment, a 
descent segment and, finally, a turn, the pilot knew in advance that it would 
be impossible for him to maintain the 3 n. mi. spacing. The pilot was told 
that his aircraft would start out "in trim" with no position or velocity 
errors. That is, the controls would initially be set for maintaining the 
conditions of airspeed and altitude required by the flight plan in the first 
region. A CDTI display, presented on the EHSI, supplied the pilot with 
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information regarding the separation between his aircraft and the lead 
aircraft. The display was updated at four second intervals and contained 
range circles at 2 and 4 n. mi. to help the pilot maintain his 3 n. mi. 

separation. Each pilot was offered a trial run before his first replication; 

the only pilot to refuse was the one who had originally made the run which 
generated the data for the target aircraft which was used in these 
experiments. A copilot was available to handle flap configuration changes 
when called for by the pilot. 

The flight plan, shown in figure 11, can be described in terms of 
regions. The aircraft was to cruise at 250 kts and 14,000 feet altitude at a 

track angle of 135° in Region 1. In Region 2, the altitude and track angle 

were to be maintained while decelerating from 250 to 200 kts airspeed. Then 
the airspeed and track angle were to be maintained while descending to 
11,000 feet in Region 3. This was followed by a period of flying at 200 kts 
and 11,000 feet in Region 4, again with the same track angle. Region 5 
required a left turn to a new track angle of 0°, and Region 6 was a straight 
run at a constant airspeed of 200 kts and a constant altitude of 11,000 feet 
to the threshold of the runway. 

The data logged for each experiment include the average and RMS values of 
the separation and its rate of change between the two aircraft in each of the 
six regions. Detailed data were recorded each half second in Regions 2 and 3 
on the pilot's aircraft and the lead aircraft for position and groundspeed. 
This detailed data can be used to describe the state trajectory in the regions 
of deceleration and descent of the flight plan. Regions 2 and 3 were selected 
for the collection of detailed data to allow for modeling to be done of the 
tracking task during deceleration and descent. Regions 1 and 4, both of which 
involve cruising at constant speed and altitude, surround the regions of 
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detailed data collection and thus allow time for the damping out of any 
transient effects associated with simulation startup or the completion of the 
simulation. Thus the state trajectory data Were available for parameter 
identification, while average and RMS data were available to allow a 
comparison between mddel -generated and experimental state trajectories in 
Regions 2 and 3. 
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RESULTS AND DISCUSSION 


The first section summarizes the results of a data analysis performed 
prior to modeling the human pilot performance of the tracking task. This 
preliminary analysis concentrates on eight response variables which are 
aggregate statistics on the task performance in Regions 2 and 3 of the flight 
plan. The response variables are the average and RMS deviations of the 
separation between the two aircraft and of the groundspeed for the pilot's 
aircraft in the two regions. The second section discusses the sensitivity of 
fit to the model parameters. The third section assesses the validity of the 
models obtained by solving the forty modeling problems. The final section 
presents a summary of the model parameter values and describes a structure 
that can be used for the general problem of modeling human pilot performance 
of the tracking task. 
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Preliminary Analysis of Data 

This part of the overall data analysis problem investigates the 
importance of the ttiree factors (pilot, control system, and replication 
number) on the response variables (average and RMS of aircraft separatidn 
distance and of the groundspeed for the pilot's aircraft for Regions 2 and 
3). The second arid third parts of the overall data analysis problem, which 
are concerned with modei verification and the analysis of model parameter 
distributions, are described irt later sections of this report. 

Let A ec jj denote the experimental average separation between the target 
and the pilot's aircraft in Region j ( j =2 ,3) for the experiments (e for 
experiments, d for separation distance), and let R edj - denote the experimental 
RMS deviation for the separation in Region j. Define A gv j and R gv j similarly 
for the experimental average and RMS deviation for the groundspeed of the 
pilot's aircraft in Region j. These eight response variables may in general 
depend on the three independent variables specifying the pilot, control 
system, and replication number. 

An analysis of variance (ANOVA, ref. 8) was performed to investigate the 
significance of the three factors upon the eight response variables. Table 1 
indicates that the type of control system is not a statistically significant 
factor for any of the eight response variables when the level of significance 
is 1 percent. Consequently, the sum of squares from the type of control 
system is pooled with the residual sum of squares in the ANOVA that follows. 

Table 2 summarizes the results for a two-way ANOVA on the eight response 
variables for Regions 2 and 3. Sums of squares from two-way interactions, 
which were not statistically significant, are pooled into the residual sum of 
squares to obtain Table 2. Table 2 shows that the pilot effect is 
statistically significant (a<.05) for six of the eight response variables. In 
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addition, four of the variables are also statistically significant at the 
a = .01 level. The table also shows that the replication number is 
statistically significant for three of the eight response variables at the 
5 percent level of significance and for one variable at the 1 percent level. 
This evidence indicates that the replication number is less important than the 
pilot in its effect on the response variables. A breakdown by replication 
number of the sample means and standard deviations is given in Table 3, and a 
breakdown by pilot is given in Table 4. Thus the preliminary analysis of the 
data indicates that any model of pilot performance must have a structure which 
can accommodate the variability of task performance with the pilot and his 
experience at performing the tracking task. 
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Sensitivity of Fit to Model Parameters 
The mathematical model has four parameters - H, t, ?, and to - which can 
be selected to cause a model state trajectory to approximate an experimental 
state trajectory. It was found from experience that the errors in the 
approximation depend strongly on the nominal spacing H. Indeed, large errors 
resulted for most of the forty experiments when H = 3.0 was fixed a priori, 
rather than having H selected to match trajectories. 

By contrast, the goodness of fit was relatively insensitive to the 
delay t in feeding back the position and velocity differences. The delay 
could in many cases be increased by four or even eight seconds without 
markedly increasing the error in the fit of the model output to the 
experimental data. The sensitivity of the fit to the parameters ? and <*> 
tended to lie between the high sensitivity to H and the low sensitivity to t. 
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Model Validation 

Each of the forty flights resulted in a trajectory which was modeled by 
selecting the parameters C, w, H, and t in the model structure 

Zq ( t) = -2£toX( t-x) - w^X(t-x), 

where X(t) depends implicitly on H. Thus there were forty different models of 
task performance, all having the same structure but each having its unique 
parameter vector ( c, w, H, x). 

Table 5 summarizes the results that were obtained. The preliminary 
analysis of data indicates that performance depends upon the pilot involved 
and his experience at performing the tracking task. This dependency can be 
modeled by regarding the parameter vector (c, w, H, x) to be a random 
selection from an event space consisting of all parameter vectors which may 
possibly result. Thus the fact that no two state trajectories will be 
identical, even for a pilot with extensive experience at performing the 
tracking task, is reflected in the model by the fact that no two parameter 
vectors selected at random are likely to be identical. 

Two major questions appear to dominate concerns regarding model 
validity. One question addresses the match of the state trajectory from a 
model to the state trajectory used to identify the parameters in that model. 
The question may be phrased as "Does the model Z Q (t) = -2?cjX(t-x) - co 2 x(t-x), 
where K, u, H, and t are chosen in consideration of the experimental data for 
a specific simulation run, yield a satisfactory fit to the experimental 
data?" This section of this report presents evidence that the model is valid 
in this sense. A second question addresses the match of the family of state 
trajectories, which result from the use of values of c, w, H, and x selected 
as random samples from distributions estimated from the data in Table 5, to 
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the family of state trajectories that will be obtained from pilots selected at 
random to perform the tracking task. This question may be phrased as "Does 

• • • a 

the model Zq ( t) = -2?coX(t-x) - urX(t-x), where c, u>, H, and x are selected at 
random from distributions based on the data in Table 5, yield trajectories 
which are representative of those which will be obtained by having human 
pilots perform the tracking task?" This question appears to require further 
research. 

The following material is concerned with the validity of the use of the 
model Zg{t)=-2Cu)X(t-x) - urX(t-x), where c, w, H, and x are chosen in 
consideration of experimental data from a specific simulation run, to fit the 
experimental data for that run. 

Graphical Comparison.- The plots in figures 12 through 15 illustrate 
graphically the errors that result. Experimental and model values are plotted 
as a function of time for both the separation and the velocity difference 
between the lead aircraft and the pilot's aircraft. Remarkably good fits were 
obtained for many of the forty experiments that were performed. For example, 
a visual inspection of figures 12 and 13 leaves little doubt that the model 
results in an excellent approximation to the separations and velocities from 
those experiments. The poorest fits include those presented in figures 14 and 
15. However, the following statistical analysis of model validity includes 
these relatively poor fits. Fortunately, the pilot and experience effects 
dominate the modeling errors even when these data are included in the 
statistical analysis. 

Statistical Analysis for Model Verification .- A statistical assessment of 
model validity is next made by comparing the response variables from the 
experimental data to their values from the models. Additional notation must 
be introduced to explain these results on model verification. As before, the 
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eight experimental response variables are A e1 *j and R e1 -j. where i=d,v (distance 
or velocity), and j=2,3 (Region 2 or 3). The modeling results in a 
corresponding set of response variables and R m jj- Each of the forty 
experiments yielded experimental values for the eight response variables A ei -j 
and R e jj. while each of the forty modeling problems yielded model output 
values for the variables A mi -j and R mi -j. 

ANOVA Comparing Pilot , Experience and Model i ng Effects .- The prel imi nary 
analysis in the first section of the Results and Discussion indicates that the 
model of pilot performance must have a structure which can accommodate 
variability of task performance with the pilot and his experience at 
performing the task. A comparison of the modeling errors with the pilot and 
experience effects can be made by performing the calculations for a three-way 
analysis of variance (pilot effect, experience effect, model data versus 
experimental data effect) for each response variable. Table 6 summarizes the 
results with two-way and higher order interactions, which were found to not be 
significant, pooled into the residual sum of squares. In Table 6, the first 
response variable refers to the average separation in Region 2 as determined 
by either the test runs or the models. For example, each of the five 
replications for the four pilots resulted in an experimentally determined 
average separation A ed2 in Region 2; each also resulted in an average 
separation A md2 determined by its model. Thus there are five replications, 
four levels for the pilots, and two levels for the source of the data (model 
versus experimental). 

The data in Table 6 shows that the modeling effect is small compared to 
the effects due to the pilot and his experience at performing the tracking 
task. Indeed, the null hypothesis that there is no modeling effect is 
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rejected for two of eight response variables when the level of significance is 
.05 and is also rejected for three of eight response variables when the level 
of significance is as high as .15. 

Student's t Test by Pairs on Modeling Errors .- Table 7 summarizes the 
results for model verification via a statistical comparison of the response 
variables. The leftmost column gives the algebraic difference in the sample 
means of the response variables. For example, this difference is 
M(A e d 2 ) = 3.218-3.228 = -0.010 n. mi. for the difference between the model 
output and the experimental results for the average separation in Region 2. 

The percentage difference, given in the middle column, is the ratio (expressed 
in percent) of the algebraic difference in the means to the mean value for the 


experiments ,J 


- M < A ed2> 

; mtw! 


x 100%. For the average separation in 


Region 2, this is -0.3%. The level of significance in Table 7 corresponds to 
a paired two-sample t test. 

The null hypothesis that the population mean for the model output equals 
the population mean for the experimental data is accepted at the .05 level of 
significance for the differences V^-Ag^, A mv2 -A ev2 , R mv2 -R e v2’ A md3- A ed3« 
R md3" R ed3 and R mv3" R ev3‘ However, the null hypothesis is rejected for the 
average velocity difference A mv 3 ~A eV 3 in Region 3 and the RMS deviation for 
separation R m d2 -R ed2 in R e 9 1on 2. The difference between the model outputs 
and experimental data for the average velocity in Region 3 has a strong 
statistical significance. Fortunately, this statistically significant 
difference may not be practically significant. The error is only two knots, 
which is an error of less than 1 percent in the mean groundspeed. Similarly, 
the statistically significant difference between the model and experimental 
data for the RMS deviation of the separation in Region 2 may also result in 
acceptably accurate approximations for many applications of the model. The 
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difference in RMS deviation for the separation is only 0.021 n. mi., or 
roughly 39 meters. This is less than a sixteen percent error of the 
experimental RMS deviation of 0.136 n. mi. 

In summary, the parameters in the model Z Q ( t) = -2onX(t-'i ) - X(t-i) can 
be selected to cause the state trajectory from the model to approximate the 
state trajectory from human pilot performance. Consequently, human 
performance of this tracking task in a MOTAS environment can be modeled by 
using a model with this structure. The problem is addressed next of selecting 
the model parameters C, w, H, and t to result in a representation of the 
variety of human responses that can reasonably be expected to occur. 
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Using the Model in a MOTAS Simulation 
These results can be applied as follows to model human performance of 
this tracking task- There was no strong evidence that the four parameters 
were statistically dependent. Consequently, the modeling information from the 
forty experiments is summarized in terms of separate distributions for H, t, 

C, and a > which are given in figures 16 through 19. Human performance of this 
tracking task can be simulated by obtaining a random parameter vector from the 
distributions of H> t , S, and w and using it to define a model which is 
integrated to obtain its state trajectory. To illustrate a method which can 
be used, assume that a CDTI-equipped aircraft with the tracking task has 
entered a MOTAS-type simulation. A model of the aircraft's performance of its 
tracking task is 

Z* 0 (t) = -2&jX(t- T ) - w 2 X(t~r) 

where H, T , £, and “ are from the distributions in figures 16 through 19. 
Assuming that R is a random sample from a uniform distribution over (0,1), a 
value of ? (for example) is determined as the value for which the cumulative 
distribution function in figure 18 equals R. In equation form, the value 
of K is chosen so that CDF ( C) = R. Random values for H, t, and oj can be 
determined similarly by using the data in figures 16, 17, and 19 respectively 
to transform an independent sample R from a uniform distribution over (0,1) 
into a random variable (H, r, orw) from a distribution having the appropriate 
shape. Thus the results can be incorporated into a model of air traffic in 
the terminal area to simulate human performance of CDTI-equipped aircraft 
performing the tracking task. 
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CONCLUDING REMARKS 


The data from forty tracking experiments have been used to model human 
performance of the tracking task. The experimental results were found to 
depend on the pilot and his experience at performing the task. The 
performance for each experiment was modeled by using a second-order 
differential equation with parameters (z, w, H, andx). A heuristic search 
was used to identify H and t, while a systematic search in ( z, w ) space was 
used for H and x given to find a local solution to the simplified 
identification problem. A family of forty parameter vectors (Z, w, H, x) 
results from solving the identification problem for the forty experiments. 

The errors resulting from the use of these parameters in modeling human pilot 
performance are acceptably small for many areas of application. Of course, 
final judgement as to model validity must be made in consideration of the 
context in which the model is to be used. If the accuracy reported herein 
appears to be unsatisfactory for a particular application, greater accuracy 
may be obtained by using different weights on the position and velocity errors 
(the weights W=(0 .5,0.5) were used), and by using different starting points 
with a smaller step size in searching over the parameter space. 

The problem of modeling human performance of this tracking task in a 
MOTAS environment goes beyond the above-mentioned problem of identifying model 
parameters to approximate a given state trajectory. Rather, one must 
determine the model parameters in a way that results in a family of state 
trajectories from the model that are representative of the family of state 
trajectories which would result from human performance. The research reported 
herein has established the validity of the model structure; in addition, a 
method has been described for translating information on the probability 
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distributions of the model parameters c, w, H, and T into specific models of 
human performance. However, additional experimentation involving other pilots 
may be required to obtain more confidence in these probability distributions 
than is allowed by basing them on only forty experiments. 
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TABLE 1. - LEVELS OF SIGNIFICANCE FOR TYPE OF CONTROL SYSTEM 


Region 2 

Regi on 

3 

Variable a 

Variable 

a 

Jed2 ’ll 

Jed3 

.49 

R ed2 

K ed3 

.27 

Jev2 

Jev3 

.28 

R e v2 

R ev3 

.05 


TABLE 2. - LEVELS OF SIGNIFICANCE FOR PILOT AND REPLICATION NUMBER 
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TABLE 3. - STATISTICS ON EIGHT RESPONSE VARIABLES BROKEN 
DOWN BY REPLICATION NUMBER 



rt ed2 

K ed2 

H ev2 

K ev2 

H ed3 

ed3 

ev3 

K ev3 

Replicate 1 
Mean 

3.46 

.176 

289. 

8.8 

3.39 

.232 

229. 

14.7 

Standard Deviation 

.17 

.077 

7. 

5.7 

.15 

.088 

4. 

6.7 

Replicate 2 
Mean 

3.26 

.134 

280. 

18.3 

3.36 

.147 

240. 

17.3 

Standard Deviation 

.09 

.015 

8. 

9.0 

.26 

.135 

6. 

21.1 

Replicate 3 
Mean 

3.33 

.143 

277. 

20.2 

3.25 

.162 

243. 

14.1 

Standard Deviation 

.26 

.037 

10. 

14.8 

.21 

.020 

11. 

9.5 

Replicate 4 
Mean 

3.22 

.133 

282. 

20.3 

3.17 

.108 

241. 

8.2 

Standard Deviation 

.15 

.062 

9. 

8.5 

.28 

.034 

7. 

2.3 

Replicate 5 
Mean 

3.19 

.145 

286. 

18.4 

3.17 

.133 

237. 

12.2 

Standard Deviation 

.19 

.075 

8. 

7.8 

.23 

.077 

5. 

10.1 

Replicate 6 
Mean 

3.21 

.145 

281. 

23.2 

3.81 

.105 

242. 

7.9 

Standard Deviation 

.21 

.070 

12. 

3.8 

.25 

.026 

8. 

5.0 

Replicate 7 
Mean 

3.16 

.139 

283. 

23.4 

3.24 

.153 

238. 

13.2 

Standard Deviation 

.24 

.067 

10. 

10.4 

.20 

.104 

8. 

8.2 

Replicate 8 
Mean 

3.14 

.163 

286. 

23.1 

3.15 

.104 

238. 

7.2 

Standard Deviation 

.16 

.090 

10. 

5.7 

.14 

.040 

7. 

2.8 

Replicate 9 
Mean 

3.17 

.085 

277. 

23.5 

3.22 

.079 

244. 

5.3 

Standard Deviation 

.20 

.018 

5. 

2.1 

.20 

.055 

6. 

2.3 

Replicate 10 
Mean 

3.15 

.101 

279. 

24.2 

3.25 

.060 

241. 

6.1 

Standard Deviation 

.10 

.025 

6. 

2.4 

.18 

.015 

4. 

5.5 

All Replications 
Mean 

3.23 

.136 

282. 

20.3 

3.24 

.128 

239. 

10.6 

Standard Deviation 

.19 

.058 

9. 

8.3 

.20 

.078 

7. 

8.9 
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TABLE 4. - STATISTICS ON EIGHT RESPONSE VARIABLES BROKEN 

DOWN BY PILOT 


^ed2 ^ed2 ^ev2 "ev2 ^ed3 ”ed3 ^ev3 "ev3 


Pilot 1 


Mean 

Standard 

Deviation 

3.05 

.18 

.103 

.032 

273. 

5. 

Pilot 2 

Mean 

Standard 

Deviation 

3.30 

.11 

.175 

.072 

286. 

9. 

Pilot 3 
Mean 

Standard 

Deviation 

3.25 

.18 

.136 

.053 

284. 

7. 

Pilot 4 

Mean 

Standard 

Deviation 

3.32 

.15 

.132 

.051 

283. 

7. 

All Pilots 
Mean 

Standard Deviation 

3.23 

.19 

.136 

.058 

282. 

9. 


21.4 

3.04 

.120 

244. 

12.6 

7.7 

.19 

.090 

7. 

5.6 

22.9 

3.40 

.186 

237. 

18.5 

9.7 

.15 

.080 

7. 

12.8 

15.0 

3.19 

• 111 

238. 

6.7 

8.2 

.13 

.068 

8. 

3.4 

22.0 

3.33 

.095 

238. 

4.6 

5.7 

.12 

.043 

6. 

2.4 

20.3 

3.24 

.128 

239. 

10.6 

8.3 

.20 

.078 

7. 

8.9 
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TABLE 5. 

- MODEL PARAMETER 

VALUES FOR 

THE FORTY 

FLIGHTS 



Control 

Replication 





Run No. 

Pilot System 

Number 

H 

T 

£ 

0) 

10 

1 

VCWS 

1 

3.60 

16.0 

0.253 

0.0264 

23 

1 

VCWS 

2 

3.25 

24.0 

0.134 

0.0261 

24 

1 

VCWS 

3 

3.20 

24.0 

0.014 

0.0300 

25 

1 

VCWS 

4 

3.00 

0.0 

0.347 

0.0384 

26 

1 

VCWS 

5 

3.00 

0.0 

0.076 

0.0374 

42 

1 

3D 

6 

3.05 

28.0 

0.249 

0.0362 

43 

1 

3D 

7 

3.05 

20.0 

0.465 

0.0322 

44 

1 

3D 

8 

3.00 

8.0 

0.092 

0.0379 

45 

1 

3D 

9 

3.00 

0.0 

0.052 

0.0387 

46 

1 

3D 

10 

3.10 

24.0 

0.076 

0.0329 

Averages 




3.13 

14.4 

0.176 

0.0336 

Standard 

Devi ations 


0.19 

11.3 

0.147 

0.0049 

30 

2 

3D 

1 

“3.30 

0.0 

D.106 

D.0279 

32 

2 

3D 

2 

3.50 

12.0 

0.117 

0.0341 

33 

2 

3D 

3 

3.45 

16.0 

0.515 

0.0316 

34 

2 

3D 

4 

3.35 

0.0 

0.315 

0.0270 

35 

2 

3D 

5 

3.50 

16.0 

0.333 

0.0270 

36 

2 

VCWS 

6 

3.45 

12.0 

0.506 

0.0257 

38 

2 

VCWS 

7 

3.55 

0.0 

0.171 

0.0249 

39 

2 

VCWS 

8 

3.20 

8.0 

0.370 

0.0270 

40 

2 

VCWS 

9 

3.40 

0.0 

0.437 

0.0394 

41 

2 

VCWS 

10 

3.40 

0.0 

0.149 

0.0332 

Averages 




3.41 

6.4 

0.302 

0.0298 

Standard 

Deviations 


0.10 

“7.1 

0.148 

0.0046 

3 

3 

3D 

1 

"3.40 

0.0 

0.179 

0.0251 

4 

3 

3D 

2 

3.35 

0.0 

0.181 

0.0276 

20 

3 

VCWS 

3 

3.80 

8.0 

-0.019 

0.0114 

21 

3 

VCWS 

4 

3.20 

0.0 

0.566 

0.0212 

22 

3 

VCWS 

5 

3.30 

0.0 

0.308 

0.0195 

47 

3 

3D 

6 

3.10 

0.0 

0.453 

0.0276 

48 

3 

3D 

7 

3.20 

8.0 

0.438 

0.0263 

49 

3 

3D 

8 

3.30 

8.0 

0.364 

0.0316 

50 

3 

VCWS 

9 

3.30 

12.0 

0.459 

0.0316 

51 

3 

VCWS 

10 

3.30 

0.0 

0.081 

0.0400 

Averages 




3.33 

3.6 

0.301 

0.0262 

Standard 

Deviations 


0.19 

4.8 

0.189 

0.0078 

13 

4 

3D 

1 

3.65 

0.0 

0.329 

“0.0228 

14 

4 

VCWS 

2 

3.45 

0.0 

0.523 

0.0277 

15 

4 

3D 

3 

3.30 

0.0 

0.292 

0.0257 

16 

4 

VCWS 

4 

3.50 

0.0 

0.007 

0.0387 

17 

4 

3D 

5 

3.30 

0.0 

0.890 

0.0230 

18 

4 

VCWS 

6 

3.45 

0.0 

-0.031 

0.0400 

19 

4 

3D 

7 

3.30 

8.0 

0.332 

0.0346 

27 

4 

3D 

8 

3.35 

0.0 

0.500 

0.0230 

28 

4 

VCWS 

9 

3.35 

0.0 

0.256 

0.0293 

29 

4 

VCWS 

10 

3.30 

0.0 

0.349 

0.0286 

Averages 




3.40 

0.8 

0.345 

0.0294 

Standard" 

Deviations 


"D.12 

2.5 

0.262 

0.0064 

Overall i 

Averages 


3.31 

6.3 

0.281 

0.0297 

Overall 

Standard Deviations 

0.19 

8.6 

0.197 

0.0064 
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TABLE 6. - COMPARISON OF PILOT EFFECT AND REPLICATION NUMBER EFFECT 

TO MODELING EFFECT 


Response 

Pilot 

Replication No. 

Modeling 

Variable 

Effect 

Effect 

Effect 


a 

a 

a 

Average Separation, Region 2 
RMS Separation, Region 2 
Average Groundspeed, Region 2 

.000 

.000 

.212 

.000 

.000 

.000 

.000 

.000 

.857 

RMS Groundspeed, Region 2 

.000 

.000 

.152 

Average Separation, Region 3 

.000 

.000 

.341 

RMS Separation, Region 3 

.000 

.000 

.087 

Average Groundspeed, Region 3 

.000 

.000 

.000 

RMS Groundspeed, Region 3 

.000 

.000 

.799 


TABLE 7. - STATISTICAL COMPARISON OF RESPONSE VARIABLES 
FROM MODEL AND EXPERIMENTS 


Response 

Variables 

Difference 

Algebraic 

in Means 
Percent 

Level of 
Significance 

a 

A md2" A ed2 

-0.01 

- 0.3% 

.176 

^md?~^ed? 

-0.02 

-15.9% 

.000 

A mv2“ A ev2 

0.05 

0.0% 

.874 

R mv2 -R ev2 

-1.06 

- 5.1% 

.138 

A md3“ A ed3 

0.01 

0.2% 

.342 

R md3“ R ed3 

-0.01 

- 7.8% 

.116 

A mv3" A ev3 

-2.00 

- 0.8% 

.001 

R mv3“ R ev3 

-0.15 

- 3.0% 

.817 
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Figure 1. - Mission Oriented Terminal Area Simulation. 






Figure 2. - Pseudo Pilot Station. 



Figure 3. - Air Traffic Control Station. 
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Figure 4. - ATOPS B-737-100 search aircraft. 
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Figure 5. - Internal arrangement of ATOPS B- 737- 100 research aircraft. 
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Roll scale (lfT) 
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Aircraft reference symbol ( 33 ) 
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Reference flight-path angle 
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Figure 8. - Electronic attitude director indicator. 
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9. - Electronic horizontal situation indicator. 
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Figure 10. - Navigation control and display unit. 
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11,000 ft. 
200 kts. 



Figure 11. - Flight plan for experiments. 
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Figure 16. - Cumulative Distribution Function for H. 



Figure 17. - Cumulative Distribution Function for x. 
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